Simulation Experiment Recipe

Objectives

The objective of this simulation experiment is to provide a toy example on how to use simChef and showcase the automated R Markdown-generated documentation. For the sake of illustration, this toy simulation experiment studies the performance of linear regression at the surface-level with the sole purpose of facilitating an easy-to-understand walkthrough.

[Typically, the objective of the simulation experiment (and this blurb) will be more scientific than instructive and will warrant additional context/background and domain knowledge.]

Data Generation

Linear Gaussian DGP

In the Linear Gaussian DGP, we simulate the feature/design matrix \(\mathbf{X} \in \mathbb{R}^{n \times p}\) from a normal distribution and the response vector \(\mathbf{y} \in \mathbb{R}^n\) from a linear model. Specifically,

\[\begin{gather*}\mathbf{X} \sim N\left(\mathbf{0}, \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}\right), \\\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon},\\\boldsymbol{\epsilon} \sim N(\mathbf{0}, \sigma^2 \mathbf{I}_n)\end{gather*}\]

Default Parameters in DGP

  • Number of samples: \(n = 200\)
  • Number of features: \(p = 2\)
  • Correlation among features: \(\rho = 0\)
  • Amount of noise: \(\sigma = 1\)
  • Coefficients: \(\boldsymbol{\beta} = (1, 0)^\top\)

[In practice, documentation of DGPs should answer the questions “what” and “why”. That is, “what” is the DGP, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]

Function

#> function(n, beta, rho, sigma) {
#>   cov_mat <- matrix(c(1, rho, rho, 1), byrow = T, nrow = 2, ncol = 2)
#>   X <- MASS::mvrnorm(n = n, mu = rep(0, 2), Sigma = cov_mat)
#>   y <- X %*% beta + rnorm(n, sd = sigma)
#>   return(list(X = X, y = y))
#> }
#> <bytecode: 0x5603191d77f8>

Input Parameters

#> $n
#> [1] 200
#> 
#> $beta
#> [1] 1 0
#> 
#> $rho
#> [1] 0
#> 
#> $sigma
#> [1] 1

Methods and Evaluation

Methods

OLS

Given some data \(\mathbf{X}\) and \(\mathbf{y}\), we fit ordinary least squares (OLS) and examine the p-values for each coefficient in the model. The p-values are computed using a two-sided t-test (see summary.lm()).

Elaborating further on the testing, we are interested in testing:

\[\begin{align*}H_0: \beta_i = 0 \quad vs. \quad H_1: \beta_i \neq 0.\end{align*}\]

To test this, we compute the observed T-statistic, defined as

\[\begin{align*}T = \frac{\hat{\beta}_i}{\hat{SE}(\hat{\beta_i})},\end{align*}\]

and then compute the two-sided p-value under the t distribution with \(n - p - 1\) degrees of freedom. If the p-value is lower than some significance value \(\alpha\), then there is sufficient evidence to reject the null hypothesis \(H_0\). Otherwise, there is not sufficient evidence, and we fail to reject the null hypothesis.

[In practice, documentation of methods should answer the questions “what” and “why”. That is, “what” is the method, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]

Function

#> function(X, y, cols = c("X1", "X2")) {
#>   lm_fit <- lm(y ~ X)
#>   pvals <- summary(lm_fit)$coefficients[cols, "Pr(>|t|)"] %>%
#>     setNames(paste(names(.), "p-value"))
#>   return(pvals)
#> }
#> <bytecode: 0x56031bdb1908>

Input Parameters

#> list()

Evaluation

Rejection Prob. (alpha = 0.1)

We define the rejection probability as the proportion of repetitions in the simulation experiment that result in a p-value \(\leq \alpha\). Here, we choose to set the significance level \(\alpha = 0.1\).

[In practice, documentation of evaluation metrics should answer the questions “what” and “why”. That is, “what” is the metric, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]

Base

Function

#> function(fit_results, alpha = 0.05) {
#>   group_vars <- c(".dgp_name", ".method_name")
#>   eval_out <- fit_results %>%
#>     dplyr::group_by(across({{group_vars}})) %>%
#>     dplyr::summarise(
#>       `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
#>       `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
#>     )
#>   return(eval_out)
#> }

Input Parameters

#> $alpha
#> [1] 0.1
Linear Gaussian DGP - Varying beta

Function

#> function(fit_results, vary_params = NULL, alpha = 0.05) {
#>   group_vars <- c(".dgp_name", ".method_name", vary_params)
#>   eval_out <- fit_results %>%
#>     dplyr::group_by(across({{group_vars}})) %>%
#>     dplyr::summarise(
#>       `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
#>       `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
#>     )
#>   return(eval_out)
#> }
#> <bytecode: 0x56031ac28060>

Input Parameters

#> $alpha
#> [1] 0.1
Linear Gaussian DGP - Varying rho

Function

#> function(fit_results, vary_params = NULL, alpha = 0.05) {
#>   group_vars <- c(".dgp_name", ".method_name", vary_params)
#>   eval_out <- fit_results %>%
#>     dplyr::group_by(across({{group_vars}})) %>%
#>     dplyr::summarise(
#>       `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
#>       `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
#>     )
#>   return(eval_out)
#> }
#> <bytecode: 0x56031bbcecc8>

Input Parameters

#> $alpha
#> [1] 0.1
Linear Gaussian DGP - Varying sigma

Function

#> function(fit_results, vary_params = NULL, alpha = 0.05) {
#>   group_vars <- c(".dgp_name", ".method_name", vary_params)
#>   eval_out <- fit_results %>%
#>     dplyr::group_by(across({{group_vars}})) %>%
#>     dplyr::summarise(
#>       `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
#>       `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
#>     )
#>   return(eval_out)
#> }

Input Parameters

#> $alpha
#> [1] 0.1

Visualizations

Power

To examine the power of the test, we plot the rejection probability as a function of \(\alpha\), that is, \(\mathbb{P}(\text{p-value} \leq \alpha)\) vs. \(\alpha\). If the coefficient is non-zero in the underlying DGP, then a larger AUC would indicate better performance in terms of the power. We will primarily focus on plotting the power of the first coefficient \(\beta_1\).

[In practice, documentation of the plotters should answer the questions “what” and “why”. That is, “what” is the plot, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]

Base

Function

#> function(fit_results, col = "X1") {
#>   plt <- ggplot2::ggplot(fit_results) +
#>     ggplot2::aes(x = .data[[paste(col, "p-value")]],
#>                  color = as.factor(.method_name)) +
#>     ggplot2::geom_abline(slope = 1, intercept = 0,
#>                          color = "darkgray", linetype = "solid", size = 1) +
#>     ggplot2::stat_ecdf(size = 1) +
#>     ggplot2::scale_x_continuous(limits = c(0, 1)) +
#>     ggplot2::labs(x = "t", y = "P( p-value \u2264 t )",
#>                   linetype = "", color = "Method")
#>   return(plt)
#> }

Input Parameters

#> list()
Linear Gaussian DGP - Varying beta

Function

#> function(fit_results, vary_params = NULL, col = "X1") {
#> 
#>   if (!is.null(vary_params)) {
#>     # deal with the case when we vary across a parameter that is vector-valued
#>     if (is.list(fit_results[[vary_params]])) {
#>       fit_results[[vary_params]] <- list_col_to_chr(fit_results[[vary_params]],
#>                                                     name = vary_params,
#>                                                     verbatim = TRUE)
#>     }
#>   }
#> 
#>   plt <- ggplot2::ggplot(fit_results) +
#>     ggplot2::aes(x = .data[[paste(col, "p-value")]],
#>                  color = as.factor(.method_name)) +
#>     ggplot2::geom_abline(slope = 1, intercept = 0,
#>                          color = "darkgray", linetype = "solid", size = 1) +
#>     ggplot2::stat_ecdf(size = 1) +
#>     ggplot2::scale_x_continuous(limits = c(0, 1)) +
#>     ggplot2::labs(x = "t", y = "P( p-value \u2264 t )",
#>                   linetype = "", color = "Method")
#>   if (!is.null(vary_params)) {
#>     plt <- plt + ggplot2::facet_wrap(~ .data[[vary_params]])
#>   }
#>   return(plt)
#> }
#> <bytecode: 0x560315970b18>

Input Parameters

#> list()
Linear Gaussian DGP - Varying rho

Function

#> function(fit_results, vary_params = NULL, col = "X1") {
#> 
#>   if (!is.null(vary_params)) {
#>     # deal with the case when we vary across a parameter that is vector-valued
#>     if (is.list(fit_results[[vary_params]])) {
#>       fit_results[[vary_params]] <- list_col_to_chr(fit_results[[vary_params]],
#>                                                     name = vary_params,
#>                                                     verbatim = TRUE)
#>     }
#>   }
#> 
#>   plt <- ggplot2::ggplot(fit_results) +
#>     ggplot2::aes(x = .data[[paste(col, "p-value")]],
#>                  color = as.factor(.method_name)) +
#>     ggplot2::geom_abline(slope = 1, intercept = 0,
#>                          color = "darkgray", linetype = "solid", size = 1) +
#>     ggplot2::stat_ecdf(size = 1) +
#>     ggplot2::scale_x_continuous(limits = c(0, 1)) +
#>     ggplot2::labs(x = "t", y = "P( p-value \u2264 t )",
#>                   linetype = "", color = "Method")
#>   if (!is.null(vary_params)) {
#>     plt <- plt + ggplot2::facet_wrap(~ .data[[vary_params]])
#>   }
#>   return(plt)
#> }
#> <bytecode: 0x56031505b8d8>

Input Parameters

#> list()
Linear Gaussian DGP - Varying sigma

Function

#> function(fit_results, vary_params = NULL, col = "X1") {
#> 
#>   if (!is.null(vary_params)) {
#>     # deal with the case when we vary across a parameter that is vector-valued
#>     if (is.list(fit_results[[vary_params]])) {
#>       fit_results[[vary_params]] <- list_col_to_chr(fit_results[[vary_params]],
#>                                                     name = vary_params,
#>                                                     verbatim = TRUE)
#>     }
#>   }
#> 
#>   plt <- ggplot2::ggplot(fit_results) +
#>     ggplot2::aes(x = .data[[paste(col, "p-value")]],
#>                  color = as.factor(.method_name)) +
#>     ggplot2::geom_abline(slope = 1, intercept = 0,
#>                          color = "darkgray", linetype = "solid", size = 1) +
#>     ggplot2::stat_ecdf(size = 1) +
#>     ggplot2::scale_x_continuous(limits = c(0, 1)) +
#>     ggplot2::labs(x = "t", y = "P( p-value \u2264 t )",
#>                   linetype = "", color = "Method")
#>   if (!is.null(vary_params)) {
#>     plt <- plt + ggplot2::facet_wrap(~ .data[[vary_params]])
#>   }
#>   return(plt)
#> }
#> <bytecode: 0x560314b76628>

Input Parameters

#> list()

Rejection Prob. (alpha = 0.1) Plot

We plot the rejection probability for \(\beta_1\) across varying parameters of the DGP to understand how characteristics of the DGP affect the test.We define the rejection probability as the proportion of repetitions in the simulation experiment that result in a p-value \(\leq \alpha\). Here, we choose to set the significance level \(\alpha = 0.1\).

[In practice, documentation of the plotters should answer the questions “what” and “why”. That is, “what” is the plot, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]

Linear Gaussian DGP - Varying beta

Function

#> function(eval_results, vary_params = NULL,
#>                                  alpha = 0.05) {
#>   eval_results <- eval_results$`Rejection Prob. (alpha = 0.1)`
#>   if (is.list(eval_results[[vary_params]])) {
#>     # deal with the case when we vary across a parameter that is vector-valued
#>     eval_results[[vary_params]] <- list_col_to_chr(eval_results[[vary_params]],
#>                                                    name = vary_params,
#>                                                    verbatim = TRUE)
#>   }
#>   plt <- ggplot2::ggplot(eval_results) +
#>     ggplot2::aes(x = .data[[vary_params]], y = `X1 Reject Prob.`,
#>                  color = as.factor(.method_name),
#>                  fill = as.factor(.method_name)) +
#>     ggplot2::labs(x = vary_params,
#>                   y = sprintf("Rejection Probability (alpha = %s)", alpha),
#>                   color = "Method", fill = "Method") +
#>     ggplot2::scale_y_continuous(limits = c(0, 1))
#>   if (is.numeric(eval_results[[vary_params]])) {
#>     plt <- plt +
#>       ggplot2::geom_line() +
#>       ggplot2::geom_point(size = 2)
#>   } else {
#>     plt <- plt +
#>       ggplot2::geom_bar(stat = "identity")
#>   }
#>   return(plotly::ggplotly(plt))
#> }
#> <bytecode: 0x5603158bf550>

Input Parameters

#> $alpha
#> [1] 0.1
Linear Gaussian DGP - Varying rho

Function

#> function(eval_results, vary_params = NULL,
#>                                  alpha = 0.05) {
#>   eval_results <- eval_results$`Rejection Prob. (alpha = 0.1)`
#>   if (is.list(eval_results[[vary_params]])) {
#>     # deal with the case when we vary across a parameter that is vector-valued
#>     eval_results[[vary_params]] <- list_col_to_chr(eval_results[[vary_params]],
#>                                                    name = vary_params,
#>                                                    verbatim = TRUE)
#>   }
#>   plt <- ggplot2::ggplot(eval_results) +
#>     ggplot2::aes(x = .data[[vary_params]], y = `X1 Reject Prob.`,
#>                  color = as.factor(.method_name),
#>                  fill = as.factor(.method_name)) +
#>     ggplot2::labs(x = vary_params,
#>                   y = sprintf("Rejection Probability (alpha = %s)", alpha),
#>                   color = "Method", fill = "Method") +
#>     ggplot2::scale_y_continuous(limits = c(0, 1))
#>   if (is.numeric(eval_results[[vary_params]])) {
#>     plt <- plt +
#>       ggplot2::geom_line() +
#>       ggplot2::geom_point(size = 2)
#>   } else {
#>     plt <- plt +
#>       ggplot2::geom_bar(stat = "identity")
#>   }
#>   return(plotly::ggplotly(plt))
#> }
#> <bytecode: 0x560314f28790>

Input Parameters

#> $alpha
#> [1] 0.1

Base Linear Regression Experiment

Rejection Prob. (alpha = 0.1)

Power

Linear Gaussian DGP

Varying beta

Rejection Prob. (alpha = 0.1)

Power

Rejection Prob. (alpha = 0.1) Plot

Parameter Values

#> $dgp
#> $dgp$`Linear Gaussian DGP`
#> $dgp$`Linear Gaussian DGP`$beta
#> $dgp$`Linear Gaussian DGP`$beta[[1]]
#> [1] 1 0
#> 
#> $dgp$`Linear Gaussian DGP`$beta[[2]]
#> [1] 1.0 0.5
#> 
#> $dgp$`Linear Gaussian DGP`$beta[[3]]
#> [1] 1 1
#> 
#> $dgp$`Linear Gaussian DGP`$beta[[4]]
#> [1] 1.0 1.5
#> 
#> $dgp$`Linear Gaussian DGP`$beta[[5]]
#> [1] 1 2
#> 
#> 
#> 
#> 
#> $method
#> list()

Varying rho

Rejection Prob. (alpha = 0.1)

Power

Rejection Prob. (alpha = 0.1) Plot

Parameter Values

#> $dgp
#> $dgp$`Linear Gaussian DGP`
#> $dgp$`Linear Gaussian DGP`$rho
#> [1] 0.0 0.2 0.5 0.9
#> 
#> 
#> 
#> $method
#> list()

Varying sigma

Rejection Prob. (alpha = 0.1)

Power

Parameter Values

#> $dgp
#> $dgp$`Linear Gaussian DGP`
#> $dgp$`Linear Gaussian DGP`$sigma
#> [1] 1 2 4 8
#> 
#> 
#> 
#> $method
#> list()
LS0tCnRpdGxlOiAiYHIgcGFyYW1zJHNpbV9uYW1lYCIKYXV0aG9yOiAiYHIgcGFyYW1zJGF1dGhvcmAiCmRhdGU6ICJgciBmb3JtYXQoU3lzLnRpbWUoKSwgJyVCICVkLCAlWScpYCIKaGVhZGVyLWluY2x1ZGVzOgogICAgLSBcdXNlcGFja2FnZXtmbG9hdH0KICAgIC0gXHVzZXBhY2thZ2V7YW1zbWF0aH0KICAgIC0gXHVzZXBhY2thZ2V7Z2Vuc3ltYn0Kb3V0cHV0OgogIHZ0aGVtZXM6OnZtb2Rlcm46CiAgICB0aHVtYm5haWxzOiBmYWxzZQpjc3M6IGNzcy9zaW1jaGVmLmNzcwpwYXJhbXM6CiAgYXV0aG9yOiAKICAgIGxhYmVsOiAiQXV0aG9yOiIKICAgIHZhbHVlOiAiIgogIHNpbV9uYW1lOgogICAgbGFiZWw6ICJTaW11bGF0aW9uIEV4cGVyaW1lbnQgTmFtZToiCiAgICB2YWx1ZTogIiIKICBzaW1fcGF0aDoKICAgIGxhYmVsOiAiUGF0aCB0byBTaW11bGF0aW9uIEV4cGVyaW1lbnQgRm9sZGVyOiIKICAgIHZhbHVlOiAiIgogIGV2YWxfb3JkZXI6CiAgICBsYWJlbDogIk9yZGVyIG9mIEV2YWx1YXRvcnM6IgogICAgdmFsdWU6IE5VTEwKICB2aXpfb3JkZXI6CiAgICBsYWJlbDogIk9yZGVyIG9mIFZpc3VhbGl6ZXJzOiIKICAgIHZhbHVlOiBOVUxMCiAgdmVyYm9zZToKICAgIGxhYmVsOiAiVmVyYm9zZSBMZXZlbDoiCiAgICB2YWx1ZTogMgotLS0KCjxzY3JpcHQgc3JjPSJqcy9zaW1jaGVmTmF2Q2xhc3MuanMiPjwvc2NyaXB0PgoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9Cm9wdGlvbnMod2lkdGggPSAxMDAwMCkKa25pdHI6Om9wdHNfY2h1bmskc2V0KAogIGVjaG8gPSBGQUxTRSwKICB3YXJuaW5nID0gRkFMU0UsCiAgbWVzc2FnZSA9IEZBTFNFLAogIGNhY2hlID0gRkFMU0UsCiAgZmlnLmFsaWduID0gImNlbnRlciIsCiAgZmlnLnBvcyA9ICJIIiwKICBmaWcuaGVpZ2h0ID0gMTIsCiAgZmlnLndpZHRoID0gMTAKKQoKb3B0aW9ucyhrbml0ci5rYWJsZS5OQSA9ICdOQScsCiAgICAgICAgZHBseXIuc3VtbWFyaXNlLmluZm9ybSA9IEZBTFNFKQoKIyBzY3JvbGxhYmxlIHRleHQgb3V0cHV0CmxvY2FsKHsKICBob29rX291dHB1dCA8LSBrbml0cjo6a25pdF9ob29rcyRnZXQoJ291dHB1dCcpCiAga25pdHI6OmtuaXRfaG9va3Mkc2V0KG91dHB1dCA9IGZ1bmN0aW9uKHgsIG9wdGlvbnMpIHsKICAgIGlmICghaXMubnVsbChvcHRpb25zJG1heC5oZWlnaHQpKSBvcHRpb25zJGF0dHIub3V0cHV0IDwtIGMoCiAgICAgIG9wdGlvbnMkYXR0ci5vdXRwdXQsCiAgICAgIHNwcmludGYoJ3N0eWxlPSJtYXgtaGVpZ2h0OiAlczsiJywgb3B0aW9ucyRtYXguaGVpZ2h0KQogICAgKQogICAgaG9va19vdXRwdXQoeCwgb3B0aW9ucykKICB9KQp9KQoKY2h1bmtfaWR4IDwtIDEKZG9jX2RpciA8LSBmaWxlLnBhdGgocGFyYW1zJHNpbV9wYXRoLCAiZG9jcyIpCmBgYAoKYGBge3IgaGVscGVyLWZ1bnN9CgojJyBHZXQgb3JkZXIgb2Ygb2JqZWN0cyB0byBkaXNwbGF5CiMnCiMnIEBwYXJhbSBvYmpfbmFtZXMgVmVjdG9yIG9mIGFsbCBvYmplY3QgbmFtZXMgdGhhdCBuZWVkIHRvIGJlIGRpc3BsYXllZC4KIycgQHBhcmFtIG9ial9vcmRlciBWZWN0b3Igb2Ygb2JqZWN0IG5hbWVzIGluIHRoZSBkZXNpcmVkIGFwcGVhcmFuY2Ugb3JkZXIuCiMnIEByZXR1cm4gVmVjdG9yIG9mIG9iamVjdCBuYW1lcyBpbiB0aGUgb3JkZXIgaW4gd2hpY2ggdGhleSB3aWxsIGJlIGRpc3BsYXllZC4KZ2V0T2JqT3JkZXIgPC0gZnVuY3Rpb24ob2JqX25hbWVzLCBvYmpfb3JkZXIgPSBOVUxMKSB7CiAgaWYgKGlzLm51bGwob2JqX29yZGVyKSkgewogICAgcmV0dXJuKG9ial9uYW1lcykKICB9IGVsc2UgewogICAgcmV0dXJuKGludGVyc2VjdChvYmpfb3JkZXIsIG9ial9uYW1lcykpCiAgfQp9CgojJyBHZXQgYWxsIGV4cGVyaW1lbnRzIHVuZGVyIGEgZ2l2ZW4gZGlyZWN0b3J5IG5hbWUKIycKIycgQHBhcmFtIGRpcl9uYW1lIG5hbWUgb2YgZGlyZWN0b3J5CiMnIEByZXR1cm4gbGlzdCBvZiBuYW1lZCBleHBlcmltZW50cwpnZXREZXNjZW5kYW50cyA8LSBmdW5jdGlvbihkaXJfbmFtZSkgewogIGV4cGVyaW1lbnRzIDwtIGxpc3QoKQogIGZvciAoZCBpbiBsaXN0LmRpcnMoZGlyX25hbWUpKSB7CiAgICBpZiAoZmlsZS5leGlzdHMoZmlsZS5wYXRoKGQsICJleHBlcmltZW50LnJkcyIpKSkgewogICAgICBpZiAoaWRlbnRpY2FsKGQsIHBhcmFtcyRzaW1fcGF0aCkpIHsKICAgICAgICBleHBfbmFtZSA8LSAiQmFzZSIKICAgICAgfSBlbHNlIHsKICAgICAgICBleHBfbmFtZSA8LSBzdHJpbmdyOjpzdHJfcmVwbGFjZV9hbGwoCiAgICAgICAgICBzdHJpbmdyOjpzdHJfcmVtb3ZlKGQsIHBhc3RlMChwYXJhbXMkc2ltX3BhdGgsICIvIikpLAogICAgICAgICAgIi8iLCAiIC0gIgogICAgICAgICkKICAgICAgfQogICAgICBleHBlcmltZW50c1tbZXhwX25hbWVdXSA8LSByZWFkUkRTKGZpbGUucGF0aChkLCAiZXhwZXJpbWVudC5yZHMiKSkKICAgIH0KICB9CiAgcmV0dXJuKGV4cGVyaW1lbnRzKQp9CgojJyBDaGVjayBpZiBleHBlcmltZW50IGV4aXN0cwojJwojJyBAcGFyYW0gZGlyX25hbWUgbmFtZSBvZiBkaXJlY3Rvcnkgb3IgdmVjdG9yIHRoZXJlb2YKIycgQHBhcmFtIHJlY3Vyc2l2ZSBsb2dpY2FsOyBpZiBUUlVFLCBjaGVja3MgaWYgZXhwZXJpbWVudCBleGlzdHMgdW5kZXIgdGhlCiMnICAgZ2l2ZW4gZGlyZWN0b3J5KHMpOyBpZiBGQUxTRSwgY2hlY2tzIGlmIGFueSBleHBlcmltZW50IGV4aXN0cyB1bmRlciB0aGUKIycgICBkaXJlY3RvcnkocykgYW5kIGl0cyBkZXNjZW5kYW50cwojJyBAcmV0dXJuIFRSVUUgaWYgZXhwZXJpbWVudCBleGlzdHMgYW5kIEZBTFNFIG90aGVyd2lzZQpleHBlcmltZW50RXhpc3RzIDwtIGZ1bmN0aW9uKGRpcl9uYW1lLCByZWN1cnNpdmUgPSBGQUxTRSkgewogIHJlcyA8LSBwdXJycjo6bWFwX2xnbChkaXJfbmFtZSwKICAgICAgICAgICAgICAgICAgICAgICAgZnVuY3Rpb24oZCkgewogICAgICAgICAgICAgICAgICAgICAgICAgIGlmICghcmVjdXJzaXZlKSB7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBleHBfZm5hbWUgPC0gZmlsZS5wYXRoKGQsICJleHBlcmltZW50LnJkcyIpCiAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4oZmlsZS5leGlzdHMoZXhwX2ZuYW1lKSkKICAgICAgICAgICAgICAgICAgICAgICAgICB9IGVsc2UgewogICAgICAgICAgICAgICAgICAgICAgICAgICAgZGVzY2VuZGFudHMgPC0gZ2V0RGVzY2VuZGFudHMoZCkKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybihsZW5ndGgoZGVzY2VuZGFudHMpID4gMCkKICAgICAgICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgICAgICAgIH0pCiAgcmV0dXJuKGFueShyZXMpKQp9CgojJyBEaXNwbGF5cyBjb250ZW50IGZvciBzcGVjaWZpZWQgcGFydCBvZiByZWNpcGUKIycKIycgQHBhcmFtIGZpZWxkX25hbWUgcGFydCBvZiByZWNpcGUgdG8gc2hvdzsgbXVzdCBiZSBvbmUgb2YgImRncCIsICJtZXRob2QiLAojJyAgICJldmFsdWF0b3IiLCBvciAidmlzdWFsaXplciIKIycgQHJldHVybiBjb250ZW50IGZvciByZWNpcGUKc2hvd1JlY2lwZVBhcnQgPC0gZnVuY3Rpb24oZmllbGRfbmFtZSA9IGMoImRncCIsICJtZXRob2QiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiZXZhbHVhdG9yIiwgInZpc3VhbGl6ZXIiKSkgewoKICBmaWVsZF9uYW1lIDwtIG1hdGNoLmFyZyhmaWVsZF9uYW1lKQogIGZ1bmNfbmFtZSA8LSBkcGx5cjo6Y2FzZV93aGVuKGZpZWxkX25hbWUgPT0gImV2YWx1YXRvciIgfiAiZXZhbCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmllbGRfbmFtZSA9PSAidmlzdWFsaXplciIgfiAidml6IiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUUlVFIH4gZmllbGRfbmFtZSkKICBkZXNjZW5kYW50cyA8LSBnZXREZXNjZW5kYW50cyhkaXJfbmFtZSA9IHBhcmFtcyRzaW1fcGF0aCkKICBvYmpzIDwtIHB1cnJyOjptYXAoZGVzY2VuZGFudHMsIH4ueFtbcGFzdGUwKCJnZXRfIiwgZmllbGRfbmFtZSwgInMiKV1dKCkpCiAgb2JqX25hbWVzIDwtIHVuaXF1ZShwdXJycjo6cmVkdWNlKHNhcHBseShvYmpzLCBuYW1lcyksIGMpKQoKICBvYmpfaGVhZGVyIDwtICI8cCBzdHlsZT0nZm9udC13ZWlnaHQ6IGJvbGQ7IGZvbnQtc2l6ZTogMjBweCc+ICVzIDwvcD4iCiAgaW52aXNfaGVhZGVyIDwtICJcblxuIyMjICVzIHsudGFic2V0IC50YWJzZXQtcGlsbHMgLnRhYnNldC1yZWNpcGUgLnRhYnNldC1jaXJjbGV9XG5cbiIKICBzaG93dHlwZV9oZWFkZXIgPC0gIlxuXG4jIyMjICVzIHsudGFic2V0IC50YWJzZXQtcGlsbHN9XG5cbiIKICBleHBfaGVhZGVyIDwtICJcblxuIyMjIyMgJXMgXG5cbiIKCiAgaWYgKGFsbChzYXBwbHkob2JqcywgbGVuZ3RoKSA9PSAwKSkgewogICAgcmV0dXJuKGNhdCgiTi9BIikpCiAgfQoKICBmb3IgKGlkeCBpbiAxOmxlbmd0aChvYmpfbmFtZXMpKSB7CiAgICBjYXQoc3ByaW50ZihpbnZpc19oZWFkZXIsICIiKSkKICAgIG9ial9uYW1lIDwtIG9ial9uYW1lc1tpZHhdCgogICAgY2F0KCI8ZGl2IGNsYXNzPSdwYW5lbCBwYW5lbC1kZWZhdWx0IHBhZGRlZC1wYW5lbCc+IikKICAgIGNhdChzcHJpbnRmKG9ial9oZWFkZXIsIG9ial9uYW1lKSkKCiAgICBjYXQoc3ByaW50ZihzaG93dHlwZV9oZWFkZXIsIGZvbnRhd2Vzb21lOjpmYSgicmVhZG1lIiwgZmlsbCA9ICJ3aGl0ZSIpKSkKICAgIHBhc3RlTWQoZmlsZS5wYXRoKGRvY19kaXIsIHBhc3RlMChmaWVsZF9uYW1lLCAicyIpLAogICAgICAgICAgICAgICAgICAgICAgcGFzdGUwKG9ial9uYW1lLCAiLm1kIikpKQoKICAgIGNhdChzcHJpbnRmKHNob3d0eXBlX2hlYWRlciwgZm9udGF3ZXNvbWU6OmZhKCJjb2RlIiwgZmlsbCA9ICJ3aGl0ZSIpKSkKICAgIGtlZXBfb2JqcyA8LSBwdXJycjo6bWFwKG9ianMsIG9ial9uYW1lKQogICAga2VlcF9vYmpzW3NhcHBseShrZWVwX29ianMsIGlzLm51bGwpXSA8LSBOVUxMCiAgICBpZiAoYWxsKHB1cnJyOjptYXBfbGdsKGtlZXBfb2JqcywKICAgICAgICAgICAgICAgICAgICAgICAgICAgfmlzVFJVRShjaGVja19lcXVhbCgueCwga2VlcF9vYmpzW1sxXV0pKSkpKSB7CiAgICAgIG9iaiA8LSBrZWVwX29ianNbWzFdXQogICAgICBjYXQoIjxiPkZ1bmN0aW9uPC9iPiIpCiAgICAgIHZ0aGVtZXM6OnN1YmNodW5raWZ5KG9ialtbcGFzdGUwKGZ1bmNfbmFtZSwgIl9mdW4iKV1dLAogICAgICAgICAgICAgICAgICBjaHVua19pZHgsIG90aGVyX2FyZ3MgPSAibWF4LmhlaWdodD0nMjAwcHgnIikKICAgICAgY2h1bmtfaWR4IDw8LSBjaHVua19pZHggKyAxCiAgICAgIGNhdCgiPGI+SW5wdXQgUGFyYW1ldGVyczwvYj4iKQogICAgICB2dGhlbWVzOjpzdWJjaHVua2lmeShvYmpbW3Bhc3RlMChmdW5jX25hbWUsICJfcGFyYW1zIildXSwKICAgICAgICAgICAgICAgICAgY2h1bmtfaWR4LCBvdGhlcl9hcmdzID0gIm1heC5oZWlnaHQ9JzIwMHB4JyIpCiAgICAgIGNodW5rX2lkeCA8PC0gY2h1bmtfaWR4ICsgMQogICAgfSBlbHNlIHsKICAgICAgZm9yIChleHAgaW4gbmFtZXMob2JqcykpIHsKICAgICAgICBvYmogPC0gb2Jqc1tbZXhwXV1bW29ial9uYW1lXV0KICAgICAgICBpZiAoaXMubnVsbChvYmopKSB7CiAgICAgICAgICBuZXh0CiAgICAgICAgfQogICAgICAgIGNhdChzcHJpbnRmKGV4cF9oZWFkZXIsIGV4cCkpCiAgICAgICAgY2F0KCI8Yj5GdW5jdGlvbjwvYj4iKQogICAgICAgIHZ0aGVtZXM6OnN1YmNodW5raWZ5KG9ialtbcGFzdGUwKGZ1bmNfbmFtZSwgIl9mdW4iKV1dLAogICAgICAgICAgICAgICAgICAgIGNodW5rX2lkeCwgb3RoZXJfYXJncyA9ICJtYXguaGVpZ2h0PScyMDBweCciKQogICAgICAgIGNodW5rX2lkeCA8PC0gY2h1bmtfaWR4ICsgMQogICAgICAgIGNhdCgiPGI+SW5wdXQgUGFyYW1ldGVyczwvYj4iKQogICAgICAgIHZ0aGVtZXM6OnN1YmNodW5raWZ5KG9ialtbcGFzdGUwKGZ1bmNfbmFtZSwgIl9wYXJhbXMiKV1dLAogICAgICAgICAgICAgICAgICAgIGNodW5rX2lkeCwgb3RoZXJfYXJncyA9ICJtYXguaGVpZ2h0PScyMDBweCciKQogICAgICAgIGNodW5rX2lkeCA8PC0gY2h1bmtfaWR4ICsgMQogICAgICB9CiAgICB9CiAgICBjYXQoIjwvZGl2PiIpCiAgfQp9CgojJyBSZWFkcyBpbiBmaWxlIGlmIGl0IGV4aXN0cyBhbmQgcmV0dXJucyBOVUxMIGlmIHRoZSBmaWxlIGRvZXMgbm90IGV4aXN0CiMnCiMnIEBwYXJhbSBmaWxlbmFtZSBuYW1lIG9mIC5yZHMgZmlsZSB0byB0cnkgcmVhZGluZyBpbgojJyBAcmV0dXJuIG91dHB1dCBvZiBmaWxlbmFtZS5yZHMgaWYgdGhlIGZpbGUgZXhpc3RzIGFuZCBOVUxMIG90aGVyd2lzZQpnZXRSZXN1bHRzIDwtIGZ1bmN0aW9uKGZpbGVuYW1lKSB7CiAgaWYgKGZpbGUuZXhpc3RzKGZpbGVuYW1lKSkgewogICAgcmVzdWx0cyA8LSByZWFkUkRTKGZpbGVuYW1lKQogIH0gZWxzZSB7CiAgICByZXN1bHRzIDwtIE5VTEwKICB9CiAgcmV0dXJuKHJlc3VsdHMpCn0KCiMnIERpc3BsYXlzIG91dHB1dCAoYm90aCBmcm9tIGV2YWx1YXRlKCkgYW5kIHZpc3VhbGl6ZSgpKSBmcm9tIHNhdmVkIHJlc3VsdHMgdW5kZXIKIycgYSBzcGVjaWZpZWQgZGlyZWN0b3J5CiMnCiMnIEBwYXJhbSBkaXJfbmFtZSBuYW1lIG9mIGRpcmVjdG9yeQojJyBAcGFyYW0gZGVwdGggaW50ZWdlcjsgZGVwdGggb2YgZGlyZWN0b3J5IGZyb20gcGFyZW50L2Jhc2UgZXhwZXJpbWVudCdzIGZvbGRlcgojJyBAcGFyYW0gYmFzZSBsb2dpY2FsOyB3aGV0aGVyIG9yIG5vdCB0aGlzIGlzIGEgYmFzZSBleHBlcmltZW50CiMnIEBwYXJhbSBzaG93X2hlYWRlciBsb2dpY2FsOyB3aGV0aGVyIG9yIG5vdCB0byBzaG93IHNlY3Rpb24gaGVhZGVyCiMnIEBwYXJhbSB2ZXJib3NlIGludGVnZXI7IDAgPSBubyBtZXNzYWdlczsgMSA9IHByaW50IG91dCBkaXJlY3RvcnkgbmFtZSBvbmx5OwojJyAgIDIgPSBwcmludCBvdXQgZGlyZWN0b3J5IG5hbWUgYW5kIG5hbWUgb2YgZXZhbHVhdG9ycy92aXN1YWxpemVycwojJyBAcmV0dXJuIGNvbnRlbnQgcmVzdWx0cyBmcm9tIGV2YWx1YXRlKCkgYW5kIHZpc3VhbGl6ZSgpIGZyb20gdGhlIGV4cGVyaW1lbnQKc2hvd1Jlc3VsdHMgPC0gZnVuY3Rpb24oZGlyX25hbWUsIGRlcHRoLCBiYXNlID0gRkFMU0UsIHNob3dfaGVhZGVyID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgICAgdmVyYm9zZSA9IDEpIHsKICBpZiAodmVyYm9zZSA+PSAxKSB7CiAgICBtZXNzYWdlKHJlcCgiKiIsIGRlcHRoKSwgYmFzZW5hbWUoZGlyX25hbWUpKQogIH0KCiAgaWYgKGRlcHRoID09IDEpIHsKICAgIGhlYWRlcl90ZW1wbGF0ZSA8LSAiXG5cbiVzICVzIHsudGFic2V0IC50YWJzZXQtcGlsbHMgLnRhYnNldC12bW9kZXJufVxuXG4iCiAgfSBlbHNlIHsKICAgIGlmIChiYXNlIHwgIWV4cGVyaW1lbnRFeGlzdHMoZGlyX25hbWUpKSB7CiAgICAgIGhlYWRlcl90ZW1wbGF0ZSA8LSAiXG5cbiVzICVzIHsudGFic2V0IC50YWJzZXQtcGlsbHN9IgogICAgfSBlbHNlIHsKICAgICAgaGVhZGVyX3RlbXBsYXRlIDwtICJcblxuJXMgJXMgey50YWJzZXQgLnRhYnNldC1waWxscyAudGFic2V0LWNpcmNsZX0iCiAgICB9CiAgfQoKICBpZiAoc2hvd19oZWFkZXIpIHsKICAgIGNhdChzcHJpbnRmKGhlYWRlcl90ZW1wbGF0ZSwKICAgICAgICAgICAgICAgIHBhc3RlKHJlcCgiIyIsIGRlcHRoKSwgY29sbGFwc2UgPSAiIiksCiAgICAgICAgICAgICAgICBiYXNlbmFtZShkaXJfbmFtZSkpKQogIH0KCiAgaWYgKGJhc2UpIHsKICAgIGNhdChwYXN0ZTAoIlxuXG4iLAogICAgICAgICAgICAgICBwYXN0ZShyZXAoIiMiLCBkZXB0aCArIDEpLCBjb2xsYXBzZSA9ICIiKSwKICAgICAgICAgICAgICAgIiBCYXNlIC0gIiwgYmFzZW5hbWUoZGlyX25hbWUpLAogICAgICAgICAgICAgICAiIHsudGFic2V0IC50YWJzZXQtcGlsbHMgLnRhYnNldC1jaXJjbGV9XG5cbiIpKQogICAgZGVwdGggPC0gZGVwdGggKyAxCiAgfQoKICBzaG93dHlwZV90ZW1wbGF0ZSA8LSBwYXN0ZTAoCiAgICAiXG5cbiIsIHBhc3RlKHJlcCgiIyIsIGRlcHRoICsgMSksIGNvbGxhcHNlID0gIiIpLCAiICVzXG5cbiIKICApCiAgZmlnbmFtZV90ZW1wbGF0ZSA8LSAiPGgzIHN0eWxlPSdmb250LXdlaWdodDogYm9sZCc+ICVzIDwvaDM+IgogIGludmlzaWJsZV9oZWFkZXIgPC0gcGFzdGUwKAogICAgIlxuXG4iLCBwYXN0ZShyZXAoIiMiLCBkZXB0aCArIDIpLCBjb2xsYXBzZSA9ICIiKSwKICAgICIgey50YWJzZXQgLnRhYnNldC1waWxsc31cblxuIgogICkKICBwbHRfdGVtcGxhdGUgPC0gcGFzdGUwKAogICAgIlxuXG4iLCBwYXN0ZShyZXAoIiMiLCBkZXB0aCArIDMpLCBjb2xsYXBzZSA9ICIiKSwgIiAlc1xuXG4iCiAgKQoKICBleHBfZm5hbWUgPC0gZmlsZS5wYXRoKGRpcl9uYW1lLCAiZXhwZXJpbWVudC5yZHMiKQogICMgZml0X2ZuYW1lIDwtIGZpbGUucGF0aChkaXJfbmFtZSwgImZpdF9yZXN1bHRzLnJkcyIpCiAgZXZhbF9mbmFtZSA8LSBmaWxlLnBhdGgoZGlyX25hbWUsICJldmFsX3Jlc3VsdHMucmRzIikKICB2aXpfZm5hbWUgPC0gZmlsZS5wYXRoKGRpcl9uYW1lLCAidml6X3Jlc3VsdHMucmRzIikKCiAgZXhwIDwtIGdldFJlc3VsdHMoZXhwX2ZuYW1lKQogICMgZml0X3Jlc3VsdHMgPC0gZ2V0UmVzdWx0cyhmaXRfZm5hbWUpCiAgZXZhbF9yZXN1bHRzIDwtIGdldFJlc3VsdHMoZXZhbF9mbmFtZSkKICB2aXpfcmVzdWx0cyA8LSBnZXRSZXN1bHRzKHZpel9mbmFtZSkKCiAgaWYgKCFpcy5udWxsKGV2YWxfcmVzdWx0cykpIHsKICAgIGNhdChzcHJpbnRmKHNob3d0eXBlX3RlbXBsYXRlLCBmb250YXdlc29tZTo6ZmEoInRhYmxlIiwgZmlsbCA9ICJ3aGl0ZSIpKSkKICAgIGV2YWxfbmFtZXMgPC0gZ2V0T2JqT3JkZXIobmFtZXMoZXZhbF9yZXN1bHRzKSwgcGFyYW1zJGV2YWxfb3JkZXIpCiAgICBmb3IgKGV2YWxfbmFtZSBpbiBldmFsX25hbWVzKSB7CiAgICAgIGlmICh2ZXJib3NlID49IDIpIHsKICAgICAgICBtZXNzYWdlKHJlcCgiICIsIGRlcHRoICsgMSksIGV2YWxfbmFtZSkKICAgICAgfQogICAgICBldmFsdWF0b3IgPC0gZXhwJGdldF9ldmFsdWF0b3JzKClbW2V2YWxfbmFtZV1dCiAgICAgIGlmIChldmFsdWF0b3IkZG9jX3Nob3cpIHsKICAgICAgICBjYXQoc3ByaW50ZihmaWduYW1lX3RlbXBsYXRlLCBldmFsX25hbWUpKQogICAgICAgIGRvLmNhbGwodnRoZW1lczo6cHJldHR5X0RULAogICAgICAgICAgICAgICAgYyhsaXN0KGV2YWxfcmVzdWx0c1tbZXZhbF9uYW1lXV0pLCBldmFsdWF0b3IkZG9jX29wdGlvbnMpKSAlPiUKICAgICAgICAgIHZ0aGVtZXM6OnN1YmNodW5raWZ5KGkgPSBjaHVua19pZHgpCiAgICAgICAgY2h1bmtfaWR4IDw8LSBjaHVua19pZHggKyAxCiAgICAgIH0KICAgIH0KICB9CgogIGlmICghaXMubnVsbCh2aXpfcmVzdWx0cykpIHsKICAgIGNhdChzcHJpbnRmKHNob3d0eXBlX3RlbXBsYXRlLAogICAgICAgICAgICAgICAgZm9udGF3ZXNvbWU6OmZhKCJjaGFydC1iYXIiLCBmaWxsID0gIndoaXRlIikpKQogICAgdml6X25hbWVzIDwtIGdldE9iak9yZGVyKG5hbWVzKHZpel9yZXN1bHRzKSwgcGFyYW1zJHZpel9vcmRlcikKICAgIGZvciAodml6X25hbWUgaW4gdml6X25hbWVzKSB7CiAgICAgIGlmICh2ZXJib3NlID49IDIpIHsKICAgICAgICBtZXNzYWdlKHJlcCgiICIsIGRlcHRoICsgMSksIHZpel9uYW1lKQogICAgICB9CiAgICAgIHZpc3VhbGl6ZXIgPC0gZXhwJGdldF92aXN1YWxpemVycygpW1t2aXpfbmFtZV1dCiAgICAgIGlmICh2aXN1YWxpemVyJGRvY19zaG93KSB7CiAgICAgICAgY2F0KGludmlzaWJsZV9oZWFkZXIpCiAgICAgICAgY2F0KHNwcmludGYoZmlnbmFtZV90ZW1wbGF0ZSwgdml6X25hbWUpKQogICAgICAgIHBsdHMgPC0gdml6X3Jlc3VsdHNbW3Zpel9uYW1lXV0KICAgICAgICBpZiAoIWluaGVyaXRzKHBsdHMsICJsaXN0IikpIHsKICAgICAgICAgIHBsdHMgPC0gbGlzdChwbHQgPSBwbHRzKQogICAgICAgIH0KICAgICAgICBpZiAoaXMubnVsbChuYW1lcyhwbHRzKSkpIHsKICAgICAgICAgIG5hbWVzKHBsdHMpIDwtIDE6bGVuZ3RoKHBsdHMpCiAgICAgICAgfQogICAgICAgIGZvciAocGx0X25hbWUgaW4gbmFtZXMocGx0cykpIHsKICAgICAgICAgIGlmIChsZW5ndGgocGx0cykgIT0gMSkgewogICAgICAgICAgICBjYXQoc3ByaW50ZihwbHRfdGVtcGxhdGUsIHBsdF9uYW1lKSkKICAgICAgICAgIH0KICAgICAgICAgIHBsdCA8LSBwbHRzW1twbHRfbmFtZV1dCiAgICAgICAgICBpZiAoaW5oZXJpdHMocGx0LCAicGxvdGx5IikgfCBpbmhlcml0cyhwbHQsICJnZyIpIHwgaW5oZXJpdHMocGx0LCAiZ2dwbG90IikpIHsKICAgICAgICAgICAgYWRkX2NsYXNzIDwtIGMoInBhbmVsIHBhbmVsLWRlZmF1bHQgcGFkZGVkLXBhbmVsIikKICAgICAgICAgIH0gZWxzZSB7CiAgICAgICAgICAgIGFkZF9jbGFzcyA8LSBOVUxMCiAgICAgICAgICB9CiAgICAgICAgICB2dGhlbWVzOjpzdWJjaHVua2lmeShwbHQsIGkgPSBjaHVua19pZHgsCiAgICAgICAgICAgICAgICAgICAgICBmaWdfaGVpZ2h0ID0gdmlzdWFsaXplciRkb2Nfb3B0aW9ucyRoZWlnaHQsCiAgICAgICAgICAgICAgICAgICAgICBmaWdfd2lkdGggPSB2aXN1YWxpemVyJGRvY19vcHRpb25zJHdpZHRoLAogICAgICAgICAgICAgICAgICAgICAgb3RoZXJfYXJncyA9ICJvdXQud2lkdGggPSAnMTAwJSciLAogICAgICAgICAgICAgICAgICAgICAgYWRkX2NsYXNzID0gYWRkX2NsYXNzKQogICAgICAgICAgY2h1bmtfaWR4IDw8LSBjaHVua19pZHggKyAxCiAgICAgICAgfQogICAgICB9CiAgICB9CiAgfQoKICBpZiAoIWlzLm51bGwoZXhwKSkgewogICAgaWYgKChsZW5ndGgoZXhwJGdldF92YXJ5X2Fjcm9zcygpJGRncCkgIT0gMCkgfAogICAgICAgIChsZW5ndGgoZXhwJGdldF92YXJ5X2Fjcm9zcygpJG1ldGhvZCkgIT0gMCkpIHsKICAgICAgY2F0KHNwcmludGYoc2hvd3R5cGVfdGVtcGxhdGUsIGZvbnRhd2Vzb21lOjpmYSgiY29kZSIsIGZpbGwgPSAid2hpdGUiKSkpCiAgICAgIGNhdCgiPGI+UGFyYW1ldGVyIFZhbHVlczwvYj4iKQogICAgICB2dGhlbWVzOjpzdWJjaHVua2lmeShleHAkZ2V0X3ZhcnlfYWNyb3NzKCksCiAgICAgICAgICAgICAgICAgIGNodW5rX2lkeCwgb3RoZXJfYXJncyA9ICJtYXguaGVpZ2h0PScyMDBweCciKQogICAgICBjaHVua19pZHggPDwtIGNodW5rX2lkeCArIDEKICAgIH0KICB9Cn0KCiMnIERpc3BsYXlzIG91dHB1dCBvZiBleHBlcmltZW50IGZvciBhbGwgb2YgaXRzIChzYXZlZCkgZGVzY2VuZGFudHMKIycKIycgQHBhcmFtIGRpcl9uYW1lIG5hbWUgb2YgcGFyZW50IGV4cGVyaW1lbnQgZGlyZWN0b3J5CiMnIEBwYXJhbSBkZXB0aCBwbGFjZWhvbGRlciBmb3IgcmVjdXJzaW9uOyBzaG91bGQgbm90IGJlIG1lc3NlZCB3aXRoCiMnIEBwYXJhbSAuLi4gb3RoZXIgYXJndW1lbnRzIHRvIHBhc3MgaW50byBzaG93UmVzdWx0cygpCnNob3dEZXNjZW5kYW50UmVzdWx0cyA8LSBmdW5jdGlvbihkaXJfbmFtZSwgZGVwdGggPSAxLCAuLi4pIHsKICBjaGlsZHJlbiA8LSBsaXN0LmRpcnMoZGlyX25hbWUsIHJlY3Vyc2l2ZSA9IEZBTFNFKQogIGlmIChsZW5ndGgoY2hpbGRyZW4pID09IDApIHsKICAgIHJldHVybigpCiAgfQogIGZvciAoY2hpbGRfaWR4IGluIDE6bGVuZ3RoKGNoaWxkcmVuKSkgewogICAgY2hpbGQgPC0gY2hpbGRyZW5bY2hpbGRfaWR4XQogICAgaWYgKCFleHBlcmltZW50RXhpc3RzKGNoaWxkLCByZWN1cnNpdmUgPSBUUlVFKSkgewogICAgICBuZXh0CiAgICB9CiAgICBpZiAoZXhwZXJpbWVudEV4aXN0cyhjaGlsZCwgcmVjdXJzaXZlID0gRkFMU0UpICYKICAgICAgICAoZXhwZXJpbWVudEV4aXN0cyhsaXN0LmRpcnMoY2hpbGQsIHJlY3Vyc2l2ZSA9IFRSVUUpWy0xXSkgfAogICAgICAgICAoZGVwdGggPT0gMSkpKSB7CiAgICAgIGJhc2UgPC0gVFJVRQogICAgfSBlbHNlIHsKICAgICAgYmFzZSA8LSBGQUxTRQogICAgfQogICAgc2hvd1Jlc3VsdHMoY2hpbGQsIGRlcHRoLCBiYXNlID0gYmFzZSwgLi4uKQogICAgc2hvd0Rlc2NlbmRhbnRSZXN1bHRzKGNoaWxkLCBkZXB0aCArIDEsIC4uLikKICB9Cn0KCgpgYGAKCiMgU2ltdWxhdGlvbiBFeHBlcmltZW50IFJlY2lwZSB7LnRhYnNldCAudGFic2V0LXZtb2Rlcm59CgojIyBPYmplY3RpdmVzIHsucGFuZWwgLnBhbmVsLWRlZmF1bHQgLnBhZGRlZC1wYW5lbH0KCmBgYHtyIG9iamVjdGl2ZXMsIHJlc3VsdHMgPSAiYXNpcyJ9CnBhc3RlTWQoZmlsZS5wYXRoKGRvY19kaXIsICJvYmplY3RpdmVzLm1kIikpCmBgYAoKIyMgRGF0YSBHZW5lcmF0aW9uCgpgYGB7ciBkZ3BzLCByZXN1bHRzID0gImFzaXMifQpzaG93UmVjaXBlUGFydChmaWVsZF9uYW1lID0gImRncCIpCmBgYAoKIyMgTWV0aG9kcyBhbmQgRXZhbHVhdGlvbgoKIyMjIE1ldGhvZHMKCmBgYHtyIG1ldGhvZHMsIHJlc3VsdHMgPSAiYXNpcyJ9CnNob3dSZWNpcGVQYXJ0KGZpZWxkX25hbWUgPSAibWV0aG9kIikKYGBgCgojIyMgRXZhbHVhdGlvbgoKYGBge3IgZXZhbHVhdG9ycywgcmVzdWx0cyA9ICJhc2lzIn0Kc2hvd1JlY2lwZVBhcnQoZmllbGRfbmFtZSA9ICJldmFsdWF0b3IiKQpgYGAKCiMjIFZpc3VhbGl6YXRpb25zCgpgYGB7ciB2aXN1YWxpemVycywgcmVzdWx0cyA9ICJhc2lzIn0Kc2hvd1JlY2lwZVBhcnQoZmllbGRfbmFtZSA9ICJ2aXN1YWxpemVyIikKYGBgCgoKCmBgYHtyIHJlcywgcmVzdWx0cyA9ICJhc2lzIn0KCiMgc2hvdyByZXN1bHRzCmlmIChleHBlcmltZW50RXhpc3RzKHBhcmFtcyRzaW1fcGF0aCkpIHsKICBjYXQoc3ByaW50ZigiXG5cbiMgQmFzZSAlcyBcblxuIiwgcGFyYW1zJHNpbV9uYW1lKSkKICBjYXQoIlxuXG4jIyB7LnRhYnNldCAudGFic2V0LXBpbGxzIC50YWJzZXQtY2lyY2xlfVxuXG4iKQogIG1lc3NhZ2Uoc3ByaW50ZigiQ3JlYXRpbmcgUiBNYXJrZG93biByZXBvcnQgZm9yICVzLi4uIiwgcGFyYW1zJHNpbV9uYW1lKSkKICBzaG93UmVzdWx0cyhwYXJhbXMkc2ltX3BhdGgsIGRlcHRoID0gMiwgYmFzZSA9IEZBTFNFLCBzaG93X2hlYWRlciA9IEZBTFNFLAogICAgICAgICAgICAgIHZlcmJvc2UgPSAwKQp9CgpzaG93RGVzY2VuZGFudFJlc3VsdHMocGFyYW1zJHNpbV9wYXRoLCB2ZXJib3NlID0gcGFyYW1zJHZlcmJvc2UpCgpgYGAK